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Executive  Summary 


Today,  organizations  that  acquire  and  build  defense  weapons  systems  are  confronted  by  nu¬ 
merous  challenges.  These  include  evolving  missions,  advancing  technologies,  volatile  fund¬ 
ing,  and  the  need  for  faster  deployment.  In  addition,  the  growing  use  of  software-intensive 
systems  (SIS)  presents  its  own  set  of  challenges.  Acquirers  and  contractors  must  assure  that 
SIS  meet  reliability,  performance,  and  interoperability  requirements  under  very  tight  time  and 
budget  constraints.  Process  improvement  methods  and  principles  can  help  acquirers  and  con¬ 
tractors  to  manage  risks  and  meet  the  challenges  they  face. 

To  determine  the  type  and  extent  of  the  benefits  that  process  improvement  methods  can  de¬ 
liver,  the  author  reviewed  technical  and  academic  literature,  examined  practitioners’  Web 
sites,  and  surveyed  process  improvement  efforts  that  have  been  applied  to  large-scale,  soft¬ 
ware-intensive  systems.  During  the  course  of  reviewing  the  articles,  documents,  books,  and 
Web  sites,  several  themes  became  apparent: 

1.  All  improvement  efforts  are  based  on  the  concept  and  the  goal  of  eliminating  errors  in 
process  upstream. 

2.  While  improvement  efforts  are  typically  driven  by  cost  and  measured  by  return  on  in¬ 
vestment,  quality  attributes  such  as  security  and  stability  are  often  of  more  value  to  the 
success  of  the  overall  project. 

3.  Most  improvement  efforts  will  yield  benefits  as  long  as  those  efforts  follow  general  rules 
and  use  appropriate  methods. 

This  special  report  documents  these  themes  by  describing  programs  and  activities  that  have 
implemented  process  improvement,  their  results  achieved,  and  in  many  cases,  the  process 
improvement  techniques  used.  Section  one  describes  the  rationale  for  surveying  the  process 
improvement  efforts  of  large-scale  SIS  programs  and  projects.  Section  two  describes  con¬ 
cepts  driving  process  improvement  and  how  they  apply  to  the  acquisition  community.  Section 
three  presents  elements  of  successful  improvement  efforts.  Section  four  describes  the  re¬ 
search  methodology  used  for  this  report.  Section  five  describes  selected  improvement  efforts. 


CMU/SEI-2004-SR-010 


v 


Abstract 


This  special  report  surveys  the  process  improvement  efforts  undertaken  by  programs  and  pro¬ 
jects  that  incorporate  software-intensive  systems.  It  describes  the  specific  process  improve¬ 
ment  efforts  undertaken  and  reports  on  the  benefits  achieved.  In  so  doing,  the  report  demon¬ 
strates  the  potential  of  process  improvement  activities  to  help  contractors  and  acquirers  meet 
their  performance,  budget,  and  delivery  time  requirements.  Additionally,  the  report  identifies 
several  recurring  themes: 

1.  All  improvement  efforts  are  based  on  the  concept  and  the  goal  of  eliminating  errors  in 
process  upstream. 

2.  While  improvement  efforts  are  typically  driven  by  cost  and  measured  by  return  on  in¬ 
vestment,  quality  attributes  such  as  security  and  stability  are  often  of  more  value  to  the 
success  of  the  overall  project. 

3.  Most  improvement  efforts  will  yield  benefits  as  long  as  those  efforts  follow  general  rules 
and  use  appropriate  methods. 

Finally,  the  report  makes  the  case  for  applying  process  improvement  methods  and  tools 
throughout  the  acquisition  effort. 
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1  Introduction 


1.1  Purpose 

This  reports  surveys  the  process  improvement  efforts  of  large-scale  programs  and  projects  that 
incorporate  software-intensive  systems.  The  report  was  developed  under  the  Army  Strategic 
Software  Improvement  Program  (ASSIP)  to  demonstrate  (a)  the  benefits  of  process  improvement 
activities,  (b)  the  rationale  for  a  coordinated  and  sustained  program  of  improvement,  and  (c)  the 
shared  characteristics  of  successful  process  improvement  efforts. 

ASSIP  is  a  long-term  program  dedicated  to  dramatically  improving  the  acquisition  of  software¬ 
intensive  systems.  The  ASSIP  is  focused  on  acquisition  programs,  people,  production,  sustain¬ 
ment,  and  institutionalizing  continuous  improvement.  Stakeholders  include  program  executive 
offices,  program  managers,  taxpayers,  contractors,  their  suppliers,  and  the  acquisition  workforce. 
The  market  customer  is  the  soldier,  the  user  of  products  delivered  to  conduct  a  military  mission. 

This  report  answers  the  fundamental  question  asked  by  many  ASSIP  stakeholders:  “Why  should  I 
invest  in  process  improvement?”  The  report  describes  various  programs  and  activities  that  have 
implemented  process  improvement,  documents  their  successes,  and,  and  in  many  cases,  identifies 
the  specific  process  improvement  techniques  used.  The  report  also  draws  several  conclusions  that 
can  be  applied  by  acquirers. 


1.2  Findings 

In  general,  military  acquisition  focuses  on  delivering  quality  weapons  systems  at  the  most  reason¬ 
able  cost.  This  mission  is  a  challenge  given  constantly  changing  threat  situations,  evolving  tech¬ 
nologies,  and  changeable  funding  streams.  Currently,  there  is  very  little  literature  on  improvement 
to  the  processes  of  acquisition  in  particular.  However,  examples  from  military  and  defense- 
contractor  sources  that  are  similar  to  the  Army’s  acquisition  efforts  suggest  that  process  im¬ 
provement  activities  can  help  acquisition  organizations  to  improve  productivity,  increase  quality, 
and  reduce  costs. 

The  processes  related  to  Army  acquisition  are  not  mysterious  or  intractable  in  relation  to  im¬ 
provement.  Within  any  system,  it  is  certain  that  efficiencies  and  improvement  opportunities  will 
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be  identified  if  an  analysis  having  integrity  and  precision  is  used.  In  every  case,  the  formula  for 
these  improvements  is  fairly  simple  and  involves  the  following  steps: 

1.  Identify  a  bottleneck,  issue,  problem,  or  opportunity. 

2.  Describe  the  context  in  which  this  issue,  problem,  or  opportunity  exists. 

3.  Identify  a  high-leverage  point  within  the  existing  state. 

4.  Clearly  describe  and  define  the  target  state  of  improvement. 

5.  Develop  the  roadmap  by  which  the  gap  between  the  current  state  and  the  desired  state  is  to 
be  closed. 

This  formula  has  been  proven  in  hundreds  of  efforts.  A  number  of  these  efforts  have  been  docu¬ 
mented  in  the  sections  that  follow. 
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2  Why  Invest  in  Process  Improvement? 


The  premise  of  this  report  is  that  improving  an  acquisition  organization’s  workforce  business 
process,  practices,  and  products  is  worthwhile  and  should  be  pursued.  Results  are  realized  as  cost 
savings,  reduced  time-to-market,  enhanced  quality,  and  improved  predictability  in  meeting  cost 
and  schedule  estimates.  The  following  sections  develop  and  support  this  premise. 


2.1  Measuring  Return  on  Investment 

Every  action  must  help  the  organization  to  achieve  its  goal.  In  business,  where  the  goal  is  profit¬ 
ability,  virtually  any  strategic  action  is  viewed  according  to  its  potential  return  on  investment 
(ROI)  computed  in  terms  of  cost  of  the  effort  relative  to  the  expected  benefit.  However,  busi¬ 
nesses  find  that  the  notion  of  ROI  goes  well  beyond  expenditures  in  dollars  because  so  many  im¬ 
portant  qualities  have  a  greater  value  than  dollars  and  cents.  In  Army  acquisition,  where  the  goal 
is  to  deliver  quality  defense  weapons  systems  at  a  reasonable  cost  as  fast  as  possible  to  the  war¬ 
fighter,  having  this  wider  notion  of  ROI  is  especially  important.  Many  military  systems  are  first 
and  foremost  mission-critical  systems.  Errors  in  the  management  and  oversight  of  the  acquisition 
of  the  system  can  result  in  the  failure  of  the  mission  and  the  loss  of  life  and  limb.  Qualities  such 
as  safety,  security,  and  system  stability  represent  “costs”  that  are  woven  into  the  fabric  of  a  sys¬ 
tem.  Yet  their  worth  far  exceeds  that  of  money. 

One  example,  the  Space  and  Naval  Warfare  Systems  Center  (SSC-SD),  was  engaged  in  develop¬ 
ing  the  SmartNet  scheduling  tool  for  High  Performance  Computing  Environments.  A  range  of 
organizations  use  this  tool  including  the  NASA  Earth  Observation  System  (EOS),  the  National 
Security  Agency,  the  USS  Coronado,  and  Florida  State  University.  SSC-SD  efforts  to  improve  its 
software  development  processes  in  order  to  achieve  Capability  Maturity  Model®  (CMM®)  Level 
3  reduced  software  change  requests  45%  over  18  months.  More  importantly,  these  efforts  led  to 
faster  software  delivery  to  users  and  better  overall  performance. 


Whether  the  ROI  is  measured  in  dollars  saved,  improvements  in  productivity,  enhanced  perform¬ 
ance,  or  other  attributes,  the  ability  of  process  improvement  to  positively  affect  change  is  clear. 
Applying  process  improvement  techniques  to  the  acquisition  community  should  yield  similar  re¬ 
sults,  thereby  helping  the  community  to  deliver  higher  quality  defense  weapons  systems  to  the 
warfighter  as  fast  as  possible  and  at  the  most  reasonable  cost. 
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2.2  Benefits  of  Process  Improvement 

The  most  significant  finding  in  this  study  is  that  genuine  improvement  efforts  will  almost  always 
result  in  dramatic,  positive  returns  on  investment  affecting  the  organization,  its  people,  practices, 
and  products.1 

In  the  case  of  the  SmartNet  effort  mentioned  in  the  previous  section,  the  standardized  language 
and  Key  Process  Areas  of  the  CMM  enabled  disparate  groups  to  work  together  toward  a  common 
improvement  goal.  The  effort  resulted  in  a  number  of  benefits  over  and  above  “faster  software 
delivery  to  users  and  better  overall  performance.”  These  additional  benefits  included  the  follow¬ 
ing 

•  better  documentation 

•  software  delivery  closer  to  scheduled  date 

•  higher  quality  software 

•  higher  customer  satisfaction 

•  improved  employee  morale 

•  better  communication  among  the  team 

•  less  overtime  required  to  get  the  job  done 

•  employees  better  educated 

•  increased  employee  pride  in  their  work  and  increased  responsibility  and  accountability  for 
their  work 

The  ability  of  process  improvement  activities  to  positively  affect  people,  practices,  and  products 
has  been  well  documented.  Ultimately,  however,  an  acquisition  organization  exists  to  serve  its 
customers.  Improving  how  an  acquirer  provides  a  product  or  service  is  not  the  same  as  delivering 
it  to  the  warfighter,  and  it  is  important  as  a  matter  of  appropriate  separation  of  concerns  to  recog¬ 
nize  that  improvement  is  an  aspect  of  the  acquirer’s  business  and  not  its  business  itself. 


2.3  Reported  Benefits  by  Project 

The  table  below  represents  highlights  of  the  larger  set  of  selected  examples.  Clearly,  no  table  can 
do  justice  to  the  stories  that  support  the  findings.  Therefore,  expanded  versions  of  some  of  these 
examples  have  been  included  in  Section  5  and  a  reference  for  each  of  the  efforts  studied  is  in¬ 
cluded  in  the  Sources  section. 


1  This  statement  requires  us  to  ask,  “What  is  genuine ?”  According  to  Phillip  Crosby,  W.  Edwards  Deming, 
and  Joseph  Juran,  a  genuine  improvement  effort  is  characterized  by  the  following  attributes 

-  active,  participating  support  of  most  senior  levels  of  management 

-  clear  definition  of  business  objectives  that  are  linked  to  the  improvement  effort 

-  careful  isolation  of  one  or,  at  most,  several  variables  for  scrutiny 

-  careful  use  of  appropriate  measures 

-  application  of  methods  to  ensure  the  integrity  of  the  measured  results 
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Table  1 :  Summary  Table  of  Improvement  Benefits 

Organ  ization  Result 


Abrams  Project 
Manager  (PM) 


Accenture 


Air  Force 
Institute  of 
Technology 


•  Empowered  the  PM  Abrams  organization  and  facilitated  M1A2  SEP  development 

•  “Leveled  the  playing”  field  with  the  contractor  by  providing  a  common  language  and  set 
of  standards 

•  Captured  the  knowledge  and  expertise  of  current  PM  Abrams  personnel  for  future  em¬ 
ployees 

•  Paved  way  for  transitioning  the  PM  Abrams  to  support  organizations  through  docu- 

mented  policies,  procedures,  and  practices _ 

Reported  5:1  benefit  as  a  ratio  of  hours  spent  on  Capability  Maturity  Model  Integration 
(CMMI)  process  activities  _ 


Cost  Performance  Index  (CPI):  The  multiple  comparison  test  showed  significant  differences 
between  CMM  Levels  1  and  2  and  between  CMM  Levels  1  and  3.  As  the  organization  in¬ 
creased  in  CMM  maturity,  its  CPI  general  approaches  LOO  with  generally  decreasing  varia- 


Boeing 

Australia 


General  Motors  • 


Schedule  Performance  Index  (SPI):  At  all  CMM  rating  levels,  the  SPI  remains  close  to  LOO. 
However  for  projects  less  than  80%  complete,  the  performance  of  Level  1  organizations  was 
consistently  below  an  SPI  of  LOO.  As  the  institute  exceeded  the  80%  completed  milestone,  it 
increased  its  SPI  to  meet  the  required  1 .00  _ 


•  Decreased  cost  to  correct  a  defect  (measured  over  18  months)  33% 

•  Decreased  time  to  deliver  releases  50  % 

•  Reduced  preparation,  conduct,  and  rework  from  pre-test  and  post-test  audits  60%  (result- 

ing  in  audits  passed  with  few  or  no  outstanding  actions). _ 

•  Increased  the  percentage  of  milestones  met  from  approximately  50%  to  approximately 


•  Decreased  the  average  of  milestones  days  late  from  approximately  50  to  fewer  than  10 
Hughes  Aircraft  Saved  $2  million  dollars  annually 

Other  improvements  include  certain  quality  of  life  attributes  such  as: 

•  Fewer  overtime  hours 


Jet  Propulsion 
Laboratory 


JP  Morgan  - 
Chase 


Litton  Data 
Systems 


•  Lower  employee  turnover 

•  Better  organizational  image _ 

Reduced  defect  identification  and  correction  time  from  as  much  as  17  hours  to  less  than  2 


•  Reduced  post-release  defects 

•  Reduced  severity  of  post-release  defects 

•  Improved  predictability  of  scheduled  deliveries _ 

Reduced  the  number  of  errors  found  during  systems  integration  30% 
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Lockheed 
Martin  MD  &  S 


•  Reduced  overhead  rate  4.5% 

•  Increased  software  productivity  30% 


Motorola 

Government 

Electronics 

Division 


Achieved  an  ROI  of  6.77  to  1  for  a  typical  100,000  Software  Lines  of  Code  (SLOC)  project. 
Such  a  project  would  span  18  months  and  require  20  software  engineers.  In  dollar  amounts, 
Motorola  saved  $61 1,200  after  investing  $90,180  in  a  typical  project. 


NASA  Goddard 

•  Reduced  cost  of  software  development  by  55% 

•  Decreased  cycle  time  by  40% 

•  Reduced  post-release  defect  rate  by  75% 

Northrop 

Grumman 

Electronics 

Systems 

Moving  from  Level  3  to  Level  4  resulted  in  20%  annual  gains  in  productivity  for  the  5 -year 
reporting  period.  This  represented  a  10%  gain  over  their  previous  annual  improvement 
achievement.  The  acceleration  resulted  in  a  $25  million  annual  cost  avoidance  over  the  5-year 

time  span. 

Northrop 
Grumman  IT 

•  Achieved  13:1  ROI  calculated  as  defects  avoided  per  hour  spent  in  training  and  defect 

prevention 

•  Met  milestones  25  times  in  a  row  (with  high  quality  and  customer  satisfaction) 

•  Increased  focus  on  quality  over  “firefighting” 

•  Earned  rating  of  “exceptional”  in  every  applicable  category  on  its  Contractor  Perform¬ 
ance  Evaluation  Survey 

Oklahoma  City 
Air  Logistics 
Center 

•  Saved  $11.3  million  over  8  years 

•  Reduced  defects  90%  from  baseline  project  to  second  project 

•  Reduced  average  cost  of  a  TPS  maintenance  action  20%  over  the  last  two  years 

•  Achieved  10X  increase  in  productivity  based  on  source  code  produced  from  baseline  to 
most  recent  project  due  to  technology  as  well  as  CMM  improvements 

Raytheon 

Northeast 

Software 

Engineering 

Center 

•  Reduced  Cost-Performance  Index  variability  by  36% 

•  Reduced  Schedule  Performance  Index  variability  by  70% 

•  Reported  productivity  gains  in  the  following  percentages: 

-  1997-1999: 30% 

-  2000: 9% 

-  2001:11% 

-  2002: 6% 

Sanchez 
Computer 
Associates,  inc. 

•  Saved  $2  million  in  first  6  months  due  to  early  detection  and  removal  of  defects 

•  Created  a  robust  training  program 

•  Applied  process  improvement  activities  to  functions  other  than  programming 

Schlumberger 

•  Reduced  validation  cycles  54% 

•  Improved  productivity  100% 

•  Increased  on-time  delivery  of  software  from  5 1  %  to  94% 

•  Reduced  post-release  defects  from  25%  to  10% 

Thales  TT&S 

Decreased  cost  and  schedule  variances  as  process  maturity  levels  increased 
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3  Elements  of  Successful  Improvement 


3.1  Common  Patterns 

Regardless  of  the  driving  need  behind  the  improvement  effort,  a  basic  pattern  appears  in  every 

review  of  successful  improvements.  This  basic  pattern  can  be  outlined  as  follows: 

1.  Establish  a  suitable  pilot  environment  in  which  changes  can  be  safely  applied. 

2.  Establish  a  very  clear  picture  of  events  and  activities  as  they  are  conducted  today  in  exacting 
terms. 

3.  Following  a  clear  description  of  what  is  happening  today,  create  an  equally  clear  picture  of 
how  the  original  needs  to  change — again,  in  very  specific  terms. 

4.  Develop  a  strategy  to  make  the  new  changes  (including  ancillaries  such  as  training  and 
modifications  to  resources). 

5.  Assess  the  effects  of  the  change  (What  was  the  ROI?). 

6.  Plan  the  next  increment. 

3.2  Significant  Findings 

This  investigation  resulted  in  the  following  findings: 

•  All  the  improvement  efforts  described  in  this  report  required  investments  in  tools,  methods, 
infrastructure,  and  behavior  change.  Culture  change  is  a  constant  in  any  substantial  organiza¬ 
tional  improvement  effort. 

•  Success  with  process  improvement  is  not  a  trivial  undertaking,  nor  does  it  happen  overnight. 
In  many  cases,  it  took  anywhere  from  two  to  five  years  for  the  changes  and  the  measurable 
benefits  to  be  realized. 

•  Success  requires  leaders  who  will  provide  environments  that  support  innovative  change,  as 
well  as  the  commitment  and  participation  of  those  involved  to  make  change  happen. 

•  Improvements  conducted  in  isolation  will  not  be  as  meaningful  without  consideration  of  the 
whole  organizational  environment. 

In  the  course  of  this  investigation,  it  also  became  apparent  that  there  are  subordinate  rules  and 

principles  that  drive  these  successful  improvement  efforts: 

•  Understand  and  articulate  the  overall  goal  for  improvement. 
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•  Gain  a  clear  picture  of  the  “lay  of  the  land” — the  context  of  the  improvement  scope  of  effort. 

•  Take  a  holistic  approach  considering  both  the  social  and  technical  aspects  of  change. 

•  Develop  a  precise  map  of  the  interlinked  processes  of  the  enterprise. 

•  Develop  a  clear  understanding  of  those  high-leverage  processes  that  will  yield  the  biggest 
“bang  for  the  buck”  when  they  are  improved. 

•  Identify  the  right  measures  with  which  to  assess  the  effects  of  changes  to  existing  processes. 

•  Proceed  carefully  and  persistently — patience  is  essential  to  long-term  improvement. 

•  Start  small,  test  a  little,  think  a  little,  test  a  little. 

Another  lesson  learned  is  that  once  the  infrastructure  for  improvement  is  created,  new  improve¬ 
ments  are  more  easily  assimilated  into  the  organization. 


3.3  Implications 

The  implications  of  the  learning  garnered  through  this  report  of  improvement  benefits  indicate 
that  starting  small  and  demonstrating  early,  measurable  successes  is  important.  Programs  may 
wish  to  tackle  known  problems  with  the  known  strategies  of  improvement.  Or  they  may  wish  to 
follow  model-based  approaches  that  have  a  more  sweeping  and  general  character,  having  in  effect 
predefined  benchmarks.  In  either  case,  the  general  pattern  of  process  description,  isolation  of  de¬ 
sired  factors  for  change,  and  clear  target  picture  remain  invariants  in  successful  improvement  ac¬ 
tivities.  These  factors  may  be  used  to  inform  and  shape  the  ASSIP  response  to  its  own  improve¬ 
ment  efforts. 
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4  Research  Method 


To  obtain  the  data  reported  here,  the  author  surveyed  34  articles  and  10  books  selected  from  200 
articles,  books,  and  resources  discovered  through  Web  and  library  searches.  The  research  covered 
a  wide  range  of  organizations,  from  prime  contractors  to  programs. 


4.1  Selection  Criteria 

Given  the  time  permitted,  the  author  applied  best  professional  judgment  in  selecting  the  reports 
shown  in  this  work.  The  criteria  for  selecting  reports  were  as  follows: 

•  acquisition-related  improvements 

•  organizational  demographic  scale 

•  use  of  sound  measurement  methods 

•  respected  information  sources 

•  plausible  claims 

•  references  from  improvement  experts 

No  particular  order  of  importance  is  implied  by  these  criteria;  each  was  considered  important  in 
its  own  right.  The  author  sought  literature  reflecting  improvement  approaches  across  many  differ¬ 
ent  kinds  of  large  and  small  commercial,  military,  and  industrial  domains  and  found  a  rich  set  of 
cases  where  many  kinds  of  improvement  strategies  were  applied  in  differing  circumstances. 

In  terms  of  the  selections  made  for  this  report,  no  particular  approach  to  improvement  was  fa¬ 
vored.  The  improvement  efforts  examined  in  this  review  were  of  many  types;  from  homegrown, 
self-designed  efforts  to  more  standard  Business  Process  Re-engineering,  Total  Quality  Manage¬ 
ment  (TQM),  and  variants.  Some  studies  were  of  single  organizations,  and  some  were  compre¬ 
hensive  studies  of  multiple  improvement  efforts.  Software  Engineering  Institute  process  literature 
was  not  favored,  nor  was  it  excluded.  Sources  are  provided  in  the  Sources  section  at  the  end  of 
this  report. 

The  criteria  used  to  select  the  documents  for  this  study  were  not  formal.  Instead,  they  represent  an 
informal  method  used  to  capture  as  many  studies  as  would  be  meaningful  and  relevant  in  as  short 
a  time  as  was  possible.  The  table  below  provides  a  thumbnail  sketch  as  to  how  the  articles  were 
categorized.  These  criteria  are  applied  here  only  to  articles  acquired  by  online  library  searches 
and  not  PowerPoint  presentations  or  books. 
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Table  2:  Categorization  ofArticies  in  this  Report 


Acquisition- 

Related 

improvement 

Organizational 

Demographic 

Scale 

Use  of 

Systematic 

Methods 

Refereed 

Source 

Cross 

Referenced 

5 

22 

26 

27 

10 

4.2  Attribute  Definitions 

Acquisition-Related  Improvement  -  Articles  in  this  category  were  those  whose  authors  served 
in  acquisition  program  offices.  Their  stories  of  improvements  came  from  their  experiences  within 
the  acquisition  environment. 


Organizational  Demographic  Scale  -  This  category  was  intended  strictly  as  a  “Kentucky 
windage”2  assessment  of  the  scope  of  the  improvement  efforts  described.  The  question  asked  by 
the  authors  was,  “Is  this  organization  similar  in  scope  and  scale  to  the  Army?”  The  corresponding 
question  might  be,  “Is  the  improvement  experience  described  in  the  report  simply  too  small  to 
provide  any  lessons  with  evidence  of  scalability  to  larger  organizations?” 

Use  of  Systematic  Methods  -  Systematic  methods  would  include  case  studies,  analytical  meth¬ 
ods  other  than  strictly  co-relational  measurement  methods,  and  more  strict  quantitative  studies. 
Articles  eliminated  from  this  category  are  opinion  pieces  and  those  dealing  with  conjecture  about 
the  achievements  that  might  be  possible.  Instead,  this  category  was  simply  a  means  of  keeping  a 
focus  upon  those  studies  whose  authors  made  a  clear  effort  to  perform  their  reporting  with  order 
and  integrity. 

Refereed  Source  -  Refereed  sources  are  those  whose  publication  is  contingent  upon  expert  re¬ 
view.  Professional  societies  and  university  publications  are  the  most  often  refereed.  Trade  maga¬ 
zines  are  most  often  not.  Articles  in  this  category  were  subject  to  expert  professional  review. 


Cross  Referenced  -  This  category  was  used  to  identify  those  articles  that  were  identified  in  at 
least  several  other  publications  related  to  improvement.  While  the  finding  of  all  of  these  articles 
indicates  that  in  a  broad  sense  they  were  all  cross-referenced  within  electronic  sources  of  some 
kind,  this  category  was  an  attempt  to  isolate  those  particular  articles  that  would  most  often  be  de¬ 
scribed  as  seminal. 


2  “Kentucky  windage”  is  an  adjustment  made  in  aiming  a  projectile  to  compensate  for  the  effects  of  wind  or 
target  movement.  It  has  the  connotation  of  being  an  adjustment  that  is  based  on  subjective  criteria  (per¬ 
sonal  experience,  judgment,  and  preference)  rather  than  on  objective  criteria  (e.g.,  gauges,  sights,  or  other 
instrumentation).  The  term  may  derive  from  the  Kentucky  Long  Rifles,  which  had  a  greater  range  than 
previously  available  firearms  and  thus  required  exaggerated  adjustments  for  the  effects  of  wind. 
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4.3  Analysis  of  Reports 

Assessing  the  benefits  of  an  improvement  program  is  not  easy.  The  sampled  set  of  studies  shown 
here  has  been  carefully  selected  in  some  measure  on  trust  in  the  integrity  of  those  organizations’ 
competence  to  conduct  these  studies  and  to  present  their  data  accurately. 

In  the  art  and  science  of  measurement,  assessing  outcomes  is  always  difficult,  especially  in  a 
highly  complex  context  where  human  beings  play  a  large  role.  The  human  laboratory  is  much 
more  difficult  to  understand  and  assess  than  an  academic  one.  Ideally,  measuring  the  return  on 
investment  for  an  improvement  effort  within  an  organization  will  follow  certain  constraints  and 
controls.  Variables  would  be  selected  and  the  context  would  be  described  in  terms  of  conflicting 
factors,  extraneous  elements,  independent  and  dependent  variables,  and  so  on.  In  real  environ¬ 
ments,  the  formalisms  of  laboratory  measurement  practices  can  be  compromised  -  sometimes 
beyond  repair. 

Many  of  these  approaches  are  summarized  in  the  following  section.  Each  summary  describes  the 
organization,  the  process  improvement  effort,  and  the  results. 
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5  Selected  Improvement  Efforts 


The  following  section  is  a  review  of  literature  on  selected  efforts  to  improve  existing  manufactur¬ 
ing,  commercial,  and  service-based  enterprises.  These  systems  span  a  variety  of  engineering  and 
manufacturing  environments,  and  include  the  following  organizations: 

•  PM  Abrams 

•  Raytheon 

•  US  Air  Force  F-16  Logistics  Operations  Division 

•  National  Aeronautics  and  Space  Administration  (NASA),  Goddard  Space  Flight  Center  DoD 

•  Data  Analysis  Center  for  Software  (DACS) 

•  Software  Engineering  Institute 

•  Space  and  Naval  Warfare  Systems  Center  (SSC-SD) 

•  Air  Force  Institute  of  Technology 

•  Motorola  Government  Electronics  Division  (GED) 

•  Northrop  Grumman  Electronics  Systems  (ES) 

These  examples  are  provided  as  a  means  of  allowing  readers  to  look  into  a  selected  sampling 
taken  from  the  wider  selection  of  documents  pertaining  to  organizational  improvement.  As  stated 
earlier,  none  of  the  examples  exactly  matches  the  specific  attributes  of  a  military  acquisition  envi¬ 
ronment.  Most  of  the  examples,  however,  come  from  environments  that  are  as  complex  as  mili¬ 
tary  acquisition  enterprises  and  are  of  a  similar  scale  in  terms  of  numbers  of  employees  and  scope 
of  operations.  In  addition,  these  examples  contain  lessons  that  are  relevant  to  the  acquisition 
community. 

Each  of  the  following  examples  contains  a  description  of  the  highlights  of  the  program,  how  it 
was  improved,  and  a  specific  lesson  learned  from  the  experience.  In  addition,  each  example  is 
accompanied  by  a  table  that  provides  the  salient  highlights  of  the  study  in  these  categories: 

•  Organizational  Study  Sponsor  -  the  organization  that  sponsored  the  improvement  effort 

•  Scope  of  Effort  -  the  size  of  the  organization  involved  in  the  improvement  effort 

•  Improvement  Focus  and  Objectives  -  the  targets  for  the  improvement  effort 

•  Method  -  the  methods  used  to  conduct  the  improvement  effort 

•  Sample  of  Reported  Benefits  -  benefits  reported  in  the  study,  cited  verbatim 

•  Cases  and  Quotations  -  quotations  and  excerpts  from  cases 

•  Sample  Sites  Described  -  information  about  sub-sites  from  an  organization  that  may  have 
been  selected  to  implement  improvement  programs;  divisional  information 
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•  Sample  of  Programs  Supported  -  connections  to  specific  military  programs  that  were  sup¬ 
ported  by  a  defense  contract  organization 

•  Period  of  Study  -  dates  in  which  the  improvement  effort  took  place 

•  Publication  Date  -  the  publication  date  of  the  study 

•  Source  Article  -  article  source  information 

5.1  PM  Abrams 

The  Abrams  M1A2  Main  Battle  Tank  is  the  workhorse  of  the  U.S.  Army  and  the  most  advanced 
ground  combat  platform  in  the  world.  The  Abrams  Ml  A2  Systems  Enhancement  Package  (SEP) 
was  designed  to  enhance  the  platform’s  mobility,  performance  and  capability.  This  advanced  digi¬ 
tal  technology  called  for  over  four  million  lines  of  code  and  eight  major  digital  computer  systems 
communicating  over  an  advanced  MIL-STD  1553  bus.3 

The  prime  contractor.  General  Dynamics  Land  Systems  Division,  had  achieved  a  Software  Ac¬ 
quisition  -  CMM  (SA-CMM)  Level  3  rating,  while  the  PM  Abrams  group  was  Level  1.  This  dif¬ 
ference  posed  a  risk  to  the  acquisition  with  respect  to  anticipated  communications  gaps  between 
acquirer  and  deliverer.  In  addition,  an  internal  report  indicated  a  50%  turnover  in  personnel  by 
2007,  representing  the  devastating  loss  of  valuable  knowledge  and  experience.  To  “level  the 
playing  field”  between  acquirer  and  deliverer  and  to  capture  valuable  knowledge  and  experience, 
PM  Abrams  set  and  achieved  its  goal  of  an  SA-CMM  Level  2  rating. 

The  Lesson:  SA-CMM  empowered  the  PM  Abrams  team.  It  facilitated  common  language  be¬ 
tween  acquirer  and  deliverer.  The  SA-CMM  assessment  experience  captured  valuable  knowledge 
and  experience  for  future  PM  Abrams  personnel  and  paved  the  way  for  transition  to  support  by 
providing  clear  documentation  and  repeatable  processes.  The  SA-CMM  experience  also  set  the 
stage  for  a  program  of  continuous  process  improvement  with  the  PM  Abrams  organization. 


Table  3:  U.S.  Army  PM  Abrams 


Organizational 

Study  Sponsor 

U.S.  Army  PM  Abrams 

Scope  of  Effort 

• 

Achieve  Software  Acquisition-Capability  Maturity  Model  Level  2  (Version  1.02)  for 
its  System  Enhancement  Package  (SEP)  project 

3  Department  of  Defense  Military  Standard  1553,  Interface  Standard  for  Digital  Time  Division  Com¬ 
mand/Response  Multiplex  Data  Bus 
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Improvement  Focus 
and  Objectives 

•  Prepare  the  organization  to  implement  the  SEP. 

•  Enable  the  organization  to  be  on  the  same  level  “playing  field”  as  the  prime 

contractor. 

•  Document  knowledge  and  expertise  of  current  project  personnel  for  future  PM 

Abrams  personnel. 

•  Document  knowledge  and  expertise  of  current  project  personnel  to  help  support 
organizations  take  responsibility  for  the  effort. 

Method 

Established  a  Software  Acquisition  Process  Team  (S  APT)  to  implement  Software 
Acquisition-Capability  Maturity  Model  Level  2  (Version  1.02) 

Sample  of  Reported 
Benefits 

Achieving  SA-CMM 

•  empowered  the  PM  Abrams  organization  and  facilitated  M1A2  SEP  develop¬ 
ment 

•  “leveled  the  playing  field”  with  the  contractor  by  providing  a  common  lan¬ 
guage  and  set  of  standards 

•  captured  the  knowledge  and  expertise  of  current  PM  Abrams  personnel  for 
future  employees 

•  paved  the  way  for  transitioning  the  PM  Abrams  SEP  to  support  organizations 
through  documented  policies,  procedures,  and  practices 

Cases  and 
Quotations 

“Without  the  strong  commitment  from  top  management,  we  probably  would  not 
have  been  able  to  attain  our  goals  in  process  improvement,”  said  authors  Col.  Don¬ 
ald  P.  Kotchman  (U.S.  Army,  David  W.  Schepke  (PM  Abrams  Tank  Systems),  Leo¬ 
nard  M.  Konwinski  (PM  Abrams),  Edward  Andres  (PM  Abrams  Tank  Systems), 

Mike  Olsem  (Science  Applications  International  Corporation),  Randy  Schneider 
(Camber  Corporation). 

Sample  Sites 
Described 

This  effort  involved  the  personnel  of  PM  Abrams  Tank  Systems,  along  with  prime 
contractor  and  systems  integrator  General  Dynamics  Land  Systems  Division,  and  its 

subcontractors. 

Sample  of 

Programs 

Supported 

The  Abrams  Ml  A2  Main  Battle  Tank  is  the  workhorse  of  the  U.S.  Army  and  the 
most  advanced  ground  combat  platform  in  the  world.  The  Project  Manager  (PM) 
Abrams  Ml  A2  Systems  Enhancement  Package  (SEP)  is  a  Pre-Planned  Product  Im¬ 
provement  (3PI)  effort. 

Period  of  Study 

1 8  months,  March  2000  -  November  2001 

Publication  Date 

2002 

Source  Article 

“Achieving  SA-CMM  Level  2  at  PM  Abrams,”  CrossTalk,  August  2002 

5.2  Raytheon 

Raytheon’s  corporate  software  improvement  effort  involved  several  organizations  (Network  Cen¬ 
tric  Systems,  Electronic  Systems,  Space  and  Airborne  Systems,  and  Missile  Systems),  and  con- 
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tinues  to  do  so  today.  The  effort  began  in  1987  with  the  adoption  of  CMM  and  has  evolved  with 
the  adoption  of  CMMI,  Six  Sigma,  and  Integrated  Product  Development  (IPD).  The  amount  of 
productivity  gain  reported  differed  according  to  the  organization  and  the  measures  it  used.  How¬ 
ever,  gains  have  continued  over  the  life  of  the  improvement  effort. 

The  Lesson:  Process  Improvement  represents  more  than  a  series  of  activities  associated  with  a 
particular  methodology.  It  represents  an  approach  to  business.  Ideally,  it  should  be  woven  into  the 
corporate  culture.  Raytheon  in  particular  has  made  the  commitment  that  CMMI  will  be  applied 
with  an  “enterprise  view,”  usable  everywhere  in  the  broadest  sense.  A  recent  example  of  the  ap¬ 
plication  of  this  philosophy  is  Raytheon’s  Learning  Institute,  where  CMMI  will  be  applied  in  an 
organization  whose  mission  is  training  and  performance  enhancement. 


Table  4:  Raytheon 


Organizational 

Study  Sponsor 

Raytheon:  Headquarters  -  Lexington,  MA;  $17  billion  defense/aerospace;  76,000 
employees  worldwide 

Scope  of  Effort 

Corporate- wide:  Multi-site  improvement 

Improvement  Focus 
and  Objectives 

Defect  prevention,  Cost-schedule  variance,  Productivity 

Method 

Multi-year  roll-out  of  Software  Capability  Maturity  Model  (CMM)  improvement 
program  followed  by  application  of  CMMI  to  advanced  business  units.  Application 
of  Six  Sigma  and  Integrated  Product  Development. 

Sample  of  Reported 
Benefits 

•  Northeast  Software  Engineering  Center:  Reduced  Cost-Performance  Index 
variability  by  36  percent.  Reduced  Schedule  Performance  Index  variability  by 

70  percent. 

•  Raytheon  NCS  reports  productivity  gains  in  the  following  percentages:  1999, 
30%;  2000, 9%;  2001, 11%;  2002,  6% 

Cases  and 
Quotations 

“Fixing  defects  early  in  the  process  is  much  cheaper  than  later  in  the  life  cycle,  and 
contributes  significantly  to  increased  productivity  and  far  less  rework.” 

-  Hope  Miller,  NCS  California  Engineering  Director 

“The  Tucson  Missile  Systems  Software  Center  improved  its  organizational  produc¬ 
tivity  by  a  factor  of  2.5x  as  it  advanced  its  software  maturity.” 

-  Dan  Nash,  Corporate  Engineering,  Director  of  Software 

Sample  Sites 
Described 

•  Raytheon  Network  Centric  Systems  -  1 1 ,000  employees;  provides  mission 
integration  for  communication  and  information  dominance,  real-time  shared 
knowledge,  and  sensor  systems.  Integrate  systems  for  all  branches  of  the  United 
States  military,  the  National  Guard,  the  Department  of  Homeland  Security,  the 
Federal  Aviation  Administration,  other  U.S.  national  security  agencies,  and  in- 

temational  customers. 

•  Raytheon  Electronics  Systems  -  50,000  employees;  business  areas  include 
reconnaissance  and  surveillance  systems;  specialized  aircraft  modification; 
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command,  control,  and  communications;  electronic  imaging;  and  other  infor¬ 
mation-based  technologies. 

•  Space  and  Airborne  Systems  -  10,000  employees;  develops  critical  space  and 
airborne  missions  programs,  e.g.,  STSS,  Global  Hawk  Sensors,  ATFLIR  Tar¬ 
geting,  APG-79  AESA  Radar,  U2  Sensors 

•  Missile  Systems  -  $3  billion  business;  design,  develop  and  produce  missile 
systems,  e.g.,  air-to  air,  strike,  surface  Navy  air  defense,  land  combat,  guided 
projectiles,  directed  energy  weapons;  products  include  AMRAAM,  Javelin,  Ex- 
calibur,  TOW,  Stinger,  Tomahawk 

Sample  of 

Programs 

Supported 

DD(X),  National  Polar-Orbiting  Environmental  Satellite  System  (NPOESS) 

Period  of  Study 

Longitudinal  report  spanning  1987  to  present 

Publication  Date 

2004 

Source  Article 

“System  of  Systems:  Raytheon  integrates  systems  &  software  processes  to  reduce 
defects,  improve  predictability,”  Software  Productivity  Consortium,  Consortium 
Quarterly,  Special  Edition,  April  2003 

5.3  U.S.  Air  Force  F-16  Logistics  Operations  Division 

The  USAF  Joint  Aeronautical  Commanders  Group  Aviation  Logistics  Board  -  Air  Force  Materiel 
Command  conducted  a  five-year  effort  (1997-2002)  to  reduce  sustainment  costs.  Under  the  effort, 
personnel  developed  a  new  method  of  defect  analysis  and  removal  named  “drill-down  analysis.” 
Drill-down  analysis  emphasizes  tracking  the  causes  of  errors  by  identifying  the  lowest-level  high- 
failure  component.  This  study  describes  the  results  of  using  this  method  in  support  of  F-16  air¬ 
craft  sustainment.  Savings  included  $43  million  in  documented  cost  containment  for  2002,  with  a 
projected  $900  million  savings  over  the  life  of  the  F-16  Program. 

The  Lesson:  Traditionally,  improvements  in  reliability  or  performance  were  considered  as  an  ex¬ 
pense,  and  were  mutually  exclusive  with  reducing  cost.  However,  the  Falcon  Flex  techniques 
consistently  reduced  cost,  increased  reliability,  and  in  one  case,  improved  performance.  In  other 
words,  the  old  ways  of  thinking  are  not  necessarily  valid,  and  improving  one  area  can  affect  an¬ 
other  area  in  positive  ways. 


Table  5:  U.S.  Air  Force  F- 16  Logistics  Operations  Division 


Organizational 

Study  Sponsor 

U.S.  Air  Force  F-16  Logistics  Operations  Division 

Scope  of  Effort 

United  States  Air  Force:  Joint  Aeronautical  Commanders  Group  Aviation  Logistics  Board 

-  Air  Force  Materiel  Command 

Improvement 

Reduced  sustainment  costs 
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Focus  and 

Objectives  _ _ 

Method  Drill-down  analysis:  Identification  of  lowest-level  high-failure  component,  followed  by 

cost  analysis  as  to  its  repair. _ 

Sample  of  •  $43  M  in  documented  sustainment  cost  savings  (2002) 

Reported  Benefits  $900  M  projected  savings  for  the  life  of  F-16  program _ 

Cases  and  . [Improvements  in  reliability  or  performance  were  considered  mutually  exclusive  with 

Quotations  reducing  cost.  ‘How  much  reliability  can  you  afford?’  was  the  common  question.  How¬ 

ever,  employment  of  the  Falcon  Flex  techniques  consistently  yields  reduced  cost,  in¬ 
creased  reliability,  and  often,  as  a  by-product,  improved  performance.” 

Sample  case:  PSP  Memory  Card 

“The  greatest  single  initiative  savings  realized  thus  far  has  been  as  the  result  of  a  PBA 
replacement  of  the  F-16  AN/APG-68  Radar  System  Programmable  Signal  Processor  (PSP) 
memory  card.  Falcon  Flex  drill-down  analysis  of  the  PSP  revealed  that  one  of  27  different 
types  of  electronic  cards  in  the  PSP,  the  memory  card,  was  responsible  for  approximately 
one  fourth  of  the  annual  PSP  maintenance  cost  of  nearly  $30,000,000.  Further,  the  analy¬ 
sis  revealed  that  the  memory  card  failure  pattern  was  distributed;  that  is,  there  was  not  one 
or  a  number  of  components  failing  that  could  provide  opportunity  for  a  reduction  in  card 
failure  frequency  by  replacing  selected  components  with  more  robust  components,  or 
through  card  modification.  Therefore,  a  PBA  at  the  card  level,  to  acquire  a  much  more 
reliable  member  card,  was  deemed  the  most  cost-effective  solution  to  reducing  the  PSP 
maintenance  cost. 

“Installation  of  the  new  memory  card  commenced  in  October  1999,  and  approximately 
80%  of  the  United  States  Air  Force  F-16  PSPs  had  been  retrofitted  by  the  end  of  FY02. 

The  number  of  PSPs  requiring  repair  has  decreased  annually  as  an  increasing  number  of 
the  new  memory  cards  have  been  installed,  directly  resulting  in  reduced  PSP  maintenance 
cost.  Realized  savings  during  the  past  three  years  actually  exceeded  projected  savings. 
Analysis  revealed  that  this  was  due  to  a  reduction  in  the  frequency  of  PSP  power  supply 
failure,  another  high-failure  PSP  assembly.  Because  the  new  memory  card  draws  less 
power  than  the  old  memory  card  it  replaced,  it  appears  that  less  stress  is  being  placed  on 
the  power  supply,  reducing  its  failure  rate.  Also,  because  the  new  memory  card  does  not 
use  one  of  the  power  supply  voltage  outputs  used  by  the  old  memory  card,  the  power  sup¬ 
ply  can  actually  suffer  a  partial  failure  and  the  PSP  function  will  be  unaffected. 

“The  net  result  of  the  PSP  memory  card  initiative  has  been  that  since  initiation  of  installa¬ 
tion  of  the  new  card,  the  reduction  in  the  quantity  of  PSPs  requiring  repair  has  saved  over 
$22  in  maintenance  cost.  The  cost  to  retrofit  the  entire  fleet  with  the  new  memory  card 
was  approximately  $5.5;  therefore,  four  times  return  on  the  investment  has  already  been 
realized.  Incidentally,  in  addition  to  being  approximately  100  times  more  reliable  than  the 
old  memory  card,  the  new  card  has  twice  the  memory  as  the  old  card,  which  enables  im¬ 
proved  PSP  performance  as  an  added  benefit. 
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Sample  Sites 
Described 

Hill  Air  Force  Base,  Ogden  Air  Logistics  Center,  Ogden,  Utah 

Sample  of 

Programs 

Supported 

Falcon  Flex  (flexible  sustainment  initiative) 

Period  of  Study 

1997  -  2002 

Publication  Date 

2003 

Source  Article 

“Falcon  Flex  -  A  Proven  F-16  Business  Practice,”  IEEE  Systems  Readiness  Technology 
Conference,  Proceedings  2003  0-7803-7837-7/03 

5.4  Goddard  Space  Flight  Center  (NASA) 

The  Software  Engineering  Laboratory  (SEL)  applied  improvement  methods  to  120  experiments 
in  process  for  the  Flight  Dynamics  Division  (FDD).  The  effort  covered  a  17-year  time  period 
(1976-1993).  The  goal  was  to  document  measurable  aspects  of  software  engineering  practice. 
However,  the  effort  also  had  more  specific  goals,  including  reducing  the  average  cycle  time  to 
produce  mission-support  software,  reduce  the  defect  rate  of  delivered  software,  and  decrease  the 
cost  of  software  required  to  support  similar  missions.  SEL  personnel  developed  and  applied 
methods  to  improve  processes.  Over  the  course  of  the  program,  the  effort  achieved  an  ROI  of  7.1 
percent  to  improvement  categories  for  model  development  (process  models),  standards  develop¬ 
ment,  results  analysis,  staff  training,  requirements  definition,  and  definition  of  experiments.  The 
$25M  cost  represented  10  percent  of  the  $250M  budget  spent  on  software  development  and  main¬ 
tenance. 

The  Lesson:  In  the  words  of  the  study’s  author,  “A  successful  process  improvement  program 
does  not  require  a  large  perturbation  or  cost  to  the  development  organization.”  Further,  the  attrib¬ 
utes  that  were  improved  in  this  case,  such  as  “staff  training,”  “reduced  cycle  time,”  “requirements 
definition,”  and  “defect  prevention”  are  applicable  to  any  kind  of  enterprise. 


Table  6:  NASA  Goddard  Space  Flight  Center 


Organizational 

Study  Sponsor 

National  Aeronautics  and  Space  Administration  (NASA),  Goddard  Space  Flight  Center 

Scope  of  Effort 

Software  Engineering  Laboratory,  history  of  120  experiments  in  process  for  the  Flight 
Dynamics  Division  (FDD) 

Improvement  Focus 
and  Objectives 

Any  measurable  aspects  of  software  engineering  practice 

Decrease  in  defect  rate  of  delivered  software 

Decrease  in  the  costs  of  software  to  support  similar  missions 

Decrease  in  the  average  cycle  time  to  produce  mission-support  software 

Measures  applied:  defects,  cost,  cycle  time 
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Sample  of  Reported 
Benefits 


Sample  Sites 
Described 


•  Created  longitudinal  baseline  of  NASA  software-intensive  engineering  projects 

•  Identified  software  engineering  measurement  constants 

•  Analyzed  the  measures  relative  to  process  changes 

•  Calculated  effort  by  phase  (e.g.,  system  test,  requirements  analysis)  -  total  hours 
reported  each  week  attributed  to  phase  currently  in  execution 

•  Calculated  effort  by  activity  (e.g.,  design,  code)  -  weekly  activity  broken  down  by 
programmer  activity  during  a  specific  project  week 

•  Created  classifications  of  error  types  (e.g.,  computational,  logic/control,  etc.) 

•  Calculated  distributions  of  effort  across  activities  and  phase  (e.g.,  Design  =  26%  on 
calendar  date  X  and  23%  overall) 

•  Documented  code  size  according  to  capability  (e.g..  Orbit/tracking  data  processing 

=  250,000  SLOC  +  percent  reused  +  development  duration  +  effort  per  calculated  in 
staff  years)  _ 


Return  on  investment  relative  to  improvement  activities  at  the  NASA  Software  Engineer¬ 
ing  Laboratory  reported  by  general  categories: 

•  Project  Overhead  =  Forms  data.  Training  (expense  =  $3M)  1 .4%  ROI 

•  Data  Processing  =  Archiving  results,  Maintaining  Database,  Interviews  (expense  = 
$6M)  2.9%  ROI 

•  Model  Development  (process  models),  Standards  Development,  Results  Analysis, 
Staff  training,  Requirements  definition,  Definition  of  experiments  (expense  = 

$15M)  7.1%  ROI  _ 


“The  total  investment  that  the  SEL  has  made  in  the  improvement  effort  has  been  ap¬ 
proximately  10  percent  of  the  total  software  development  cost  in  the  FDD.  Project  over¬ 
head  represents  costs  incurred  due  to  developers  attending  training  (in  new  processes), 
completing  data  collection  forms,  participating  in  interviews,  and  providing  detailed 
additional  information  requested  by  the  analysts.  This  overhead  for  data  collection  and 
process  change  is  extremely  small.  It  is  now  nearly  impossible  to  measure  except  in  the 
cases  of  very  large  process  changes,  such  as  using  a  new  language  (longer  training,  meet¬ 
ings,  etc.).  For  projects  participating  in  the  routine  process  improvement  efforts,  the  im¬ 
pact  is  less  than  1  percent  of  the  total  software  cost.  A  successful  process  improvement 
program  does  not  require  a  large  perturbation  or  cost  to  the  development  organization.” 

“SEL  activities  have  cost  approximately  $25  million  in  an  organization  that  has  spent 
approximately  $250  million  on  software  development  and  maintenance.  There  is  no  em¬ 
pirical  evidence  to  indicate  whether  this  1:10  ratio  would  persist  when  process  improve¬ 
ment  is  expanded  to  a  larger  organization.  Very  preliminary  data  from  SEL  experience  in 
carrying  out  a  broader  NASA-wide  program  indicate  that  project  overhead  would  remain 
low,  and  that  the  data  repository  could  function  with  15  persons  to  support  development 
organizations  that  are  orders  of  magnitude  larger  than  the  FDD/’ _ 


Software  Engineering  Laboratory  -  300  employees 
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Sample  of  Programs 
Supported 

Flight  Dynamics  Division  -  System  sizes  range  from  10k  -  1.5M  SLOC 

Period  of  Study 

1976-  1993 

Publication  Date 

1994 

Source  Article 

“Software  Process  Improvement  in  the  NASA  Software  Engineering  Laboratory,”  SEI 
Technical  Report,  December  1994,  CMU/SEI-94-TR-22,  ESC-TR-94-022 

5.5  DoD  Data  Analysis  Center  for  Software  (D ACS) 

The  DoD  Data  Analysis  Center  (DACS)  conducted  an  extensive  literature  review  of  improvement 
efforts  from  1976  to  1994.  As  part  of  this  review,  DACS  personnel  examined  the  work  of  Boeing 
Space  Transportation  Systems,  Hewlett-Packard,  Schlumberger,  and  other  organizations.  The 
study  looked  for  measured  defect  identification  and  removal,  reduced  rework,  increased  produc¬ 
tivity,  decreased  cycle  time,  and  other  factors.  Significantly,  many  of  the  organizations  reported 
improvements  in  several  areas.  For  example,  Schlumberger  reported  54%  fewer  validation  cy¬ 
cles,  a  51%  increase  in  on-time  delivery,  and  a25%  reduction  in  post-release  defects. 


The  Lesson:  Software  and  system  improvement  activities  do  not  exist  in  isolation.  Rather,  the 
results  of  these  activities  typically  have  a  domino  effect,  improving  performance  and  reducing 
costs  throughout  the  entire  development  cycle.  Again,  the  dimensions  of  improvement  in  these 
studies;  defect  prevention,  increased  productivity,  decreased  cycle  time,  and  reduced  rework — are 
pertinent  to  any  enterprise. 


Table  7:  DoD  Data  Analysis  Center  for  Software 


Organizational 

Study  Sponsor 

DoD  Data  Analysis  Center  for  Software  (DACS) 

Scope  of  Effort 

Department  of  Defense 

Improvement  Focus 
and  Objectives 

•  Defect  identification  and  removal 

•  Reduced  rework 

•  Increased  productivity 

•  Decreased  cycle  time 

•  Increased  software  reuse 

•  Requirements  management 

Method 

General  literature  review  of  known  studies  on  the  efficacy  of  improvement  efforts 

Sample  of  Reported 
Benefits 

•  54%  Fewer  validation  cycles  (Schlumberger) 

•  100%  Productivity  Increase  (Schlumberger) 

•  5 1  %  Increase  in  on-time  delivery  (Schlumberger) 

•  25%  Post-release  defects  (Schlumberger) 

•  Most  defects  eliminated  before  testing  (BOEING  STS) 
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Cases  and 
Quotations 


Sample  Sites 
Described 


•  31  %  Reduced  rework  7.75 : 1  ROI  (BOEING  STS) 

•  55%  Reduced  software  development  costs  (NASA  SEL) 

•  40%  Decreased  cycle  time  (NASA  SEL) 

•  75%  Reduction  in  post-release  defect  rate  (NASA  SEL) 

•  5 1  %  Reduction  in  defects  (per  KSLOC)  (Hewlett-Packard) 

•  57%  Increase  in  productivity  (Hewlett-Packard) _ _ _ _ 

“O’Connor  (1994)  described  Rockwell  International’s  reuse  experience  with  command 
and  control  systems.  They  used  the  Synthesis  Methodology,  as  developed  by  the  Soft¬ 
ware  Productivity  Consortium  (SPC).  Reuse  is  integral  to  this  methodology  which  forms 
the  foundation  of  a  product  family  and  associated  production  processes.  It  defines  a  sys¬ 
tematic  approach  to  identifying  the  commonalities  and  variabilities  necessary  to  charac¬ 
terize  a  standardized  product  line  as  a  domain.  A  domain  is  a  product  family  and  process 
for  producing  instances  of  that  family.  Commonalities  reflect  work  that  can  be  accom¬ 
plished  once  and  reused  to  create  any  product.  Variability  specifies  compile  time  parame¬ 
ters  to  customize  the  application.  O’Connor  stated  that  the  cost  of  creating  a  reuse  do¬ 
main  is  approximately  equivalent  to  the  cost  of  making  a  hand-crafted  system  in  that 
domain  of  the  same  size.  Although  no  specific  savings  were  noted,  the  author  did  state 
that  the  benefits  of  this  reuse  methodology  were  improved  productivity,  improved  prod¬ 
uct  quality,  improved  responsiveness  to  customer  needs,  lower  bids  on  projects,  and  in¬ 
stitutionalization  of  shared  knowledge  and  expertise  among  systems  in  a  business  area.” 

“In  a  seven-year  period  the  NASA  Software  Engineering  Laboratory  (Basili,  1994  and 
McGarry  1993)  has  increased  the  average  level  of  reuse  by  300%  from  -20%  to  nearly 
79%.  At  the  same  time  the  error  rate  has  decreased  by  75%  from  4.5  to  1  error/KSLOC, 
and  the  cost  of  software  per  mission  has  decreased  by  55%  from  -490  staff  months  to 
-210  staff  months.” 

“Yamamura  and  Wigle  (1997)  state  that  Boeing’s  improvement  efforts  resulted  in  excel¬ 
lent  performance,  high  customer  satisfaction,  satisfied  employees,  and  a  97%  to  100% 
award  fee  for  6  years.  The  authors  state  that  employee  satisfaction  grew  from  74%  to 
96%  because  of  the  improvements.  They  state  that  employees  take  pride  in  their  accom¬ 
plishments  as  they  dramatically  reduce  defects.” 

“Brodman  and  Johnson  (1995)  discuss  ‘spillover’  benefits  from  process  improvement, 
including  improved  morale  and  confidence  of  developers,  less  overtime,  less  employee 
turnover,  and  improved  competitive  advantage.  The  authors  describe  how  increased  pro¬ 
ductivity  can  mean  a  more  competitive  edge  in  bidding  on  contracts,  and  can  increase  the 
company’s  capacity  to  do  work  and  thus  perform  more  work  within  a  given  period  of 
time.  Meeting  schedule  and  cost  projections  can  translate  to  customer  satisfaction,  repeat 
business,  and  decreased  time  to  market  and  improved  product  quality  translates  to  more 
dollars  on  the  bottom  line.”  _ 


•  NASA 

•  Boeing  Space  Transportation  Systems 

•  Motorola  _ 
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Tinker  AFB 


•  Hughes  Aircraft 

•  Litton  Data  Systems 

•  Jet  Propulsion  Laboratory 

•  Lockheed  Martin 

•  Hewlett-Packard 

•  Cardiac  Pacemakers  Incorporated 

•  Bull  HN  Information  Systems 

•  Shell  Research  -  Seismic  Software  Support  Center 

•  Raytheon 

•  Schlumberger 

Sample  of  Programs 

•  Air  Force  Materiel  Command 

Supported 

•  Several  NASA  programs  reported  in  earlier  section 

•  Space  Shuttle 

•  Future  Combat  Systems 

•  Joint  Tactical  Radio  System 

•  C-17  Airlifter 

•  F-15  Tactical  Fighter 

•  Evolved  Expendable  Launch  Vehicle 

•  Unmanned  Combat  Air  Vehicle 

Period  of  Study 

1976  -  present 

Publication  Date 

1994 

Source  Article 

“A  Business  Case  for  Software  Process  Improvement  Revised:  Measuring  Return  on 
Investment  from  Software  Engineering  and  Management,”  A  DACS  State-of-the-Art 
Report,  Air  Force  Research  Laboratory  -  Information  Directorate  (AFRL/IF)  525 

Brooks  Road,  Rome,  NY  13441-4505,  September  1999 

5.6  Software  Engineering  Institute 

The  SEI  evaluated  the  software  improvement  activities  of  12  medium-  to  large-scale  industrial, 
commercial,  or  defense  industry  organizations,  including  Accenture,  General  Motors,  Thales  Re¬ 
search  and  Technology,  and  Sanchez  Computer  Associates.  The  meta-study  looked  at  implemen¬ 
tations  of  CMMI  or  Software  Capability  Maturity  Model  (SW-CMM).  In  addition  to  the  expected 
benefits  of  improved  productivity,  and  reduced  rework,  the  organizations  also  improved  their 
ability  to  meet  milestones  and  reduce  variances  in  predicted  cost  and  delivery  schedules. 

The  Lesson:  The  improvement  activities  helped  the  organizations  better  predict  cost,  schedule, 
and  delivery.  In  some  cases,  those  improved  forecasting  capabilities  may  be  worth  more  to  ac- 
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quirers  and  contractors  than  the  associated  cost  savings.  Once  again,  the  specific  dimensions,  hav¬ 
ing  largely  to  do  with  reduction  of  processing  time  errors  apply  to  any  domain  of  activity. 


Table  8:  Software  Engineering  Institute 


Organizational 
Study  Sponsor 


Scope  of  Effort 


Software  Engineering  Institute 


12  medium-  to  large-scale  industrial,  commercial  or  defense  industry  organizations  exam¬ 
ined  in  regards  to  improvement  efforts  _ _ _ 


Improvement 
Focus  and 
Objectives 

Method 

Sample  of 
Reported  Benefits 


Implementation  of  CMMI  or  Software  Capability  Maturity  Model  (SW-CMM) 


Meta-study  of  the  application  of  CMMI/SW-CMM  in  medium  and  large  organizations 


Organization 

Improvement  Description 

Category 

Lockheed  Martin 

M&DS 

4.5%  decline  in  overhead  rate 

Cost 

Northrop  Grum¬ 
man  IT 

Improved  and  stabilized  Cost  Performance 

Index 

Cost 

Thales  Research 

and  Technology 

20%  reduction  in  average  cost  variance 

Cost 

Thales  Research 

and  Technology 

60%  reduction  in  cost  of  customer  acceptance 

Cost 

Thales  Training 

and  Simulation 

Cost  variance  decreased  as  process  maturity 

increased 

Cost 

Boeing,  Australia 

Reduced  by  half  the  amount  of  time  required  to 

turn  around  releases 

Schedule 

Boeing,  Australia 

60%  reduction  in  work  and  fewer  outstanding 
actions  following  pre-test  and  post-test  audits 

Schedule 

i  General  Motors 

|  i 

1 

Increased  the  percentage  of  milestones  met 
from  approximately  50%  to  approximately  95% 

Schedule 

|  General  Motors 

|  Decrease  the  average  of  milestones  days  late 
from  approximately  50  to  fewer  than  10 

Schedule 

!  JP  Morgan  Chase 

|  Increased  through-put  resulting  in  more  releases 

per  year 

! 

I  Schedule 

L  .  . .  .  ....  J 

Lockheed  Martin 
!  M&DS 

30%  increase  in  software  productivity 

!  i 

i  Schedule  j 

j 

|  Northrop  Grum- 

_  Y  rn 

Improved  and  stabilized  Schedule  Performance 

t _ i _ 

Schedule 
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Northrop  Gram-  Met  every  milestone  (25  in  a  row)  on  time,  with  Schedule 

;  man  IT  ;  high  quality  and  customer  satisfaction 

I  Bosch  Gasoline 

I  Systems 

10%  improvement  in  first  pass  yield  leading  to 

reduction  in  rework 

Schedule 

1 

i  1 

Bosch  Gasoline  j 

Systems 

15%  improvement  in  internal  on-time  delivery 

t 

Schedule 

JP  Morgan  Chase 

Improved  predictability  of  delivery  schedule 

Schedule 

Thales  Training 

and  Simulation 

Schedule  variances  decreased  as  process  matur¬ 
ity  increased 

Schedule 

Northrop  Grum¬ 
man  IT 

Only  2%  of  all  defects  found  in  the  fielded 

system 

Quality 

Northrop  Grum¬ 
man  IT 

Reduction  in  defects  found  from  6.6  per  KLOC 
to  2.1  over  5  causal  analysis  cycles 

Quality 

Northrop  Grum¬ 
man  IT 

Increased  focus  on  quality  by  developers 

Quality 

Bosch  Gasoline 

Systems 

Reduction  in  error  cases  in  the  factory  by  one 

order  of  magnitude 

Quality 

JP  Morgan  Chase 

Reduction  in  number  and  severity  of  post¬ 
release  defects 

Quality 

Sanchez  Computer 

Associates,  Inc. 

Improved  quality  of  code 

Quality 

Northrop  Grum¬ 
man  IT 

Received  more  than  98%  of  possible  customer 

award  fees 

Customer 

Satisfaction 

Northrop  Grum¬ 
man  IT 

Earned  a  rating  of  “Exceptional”  in  every  appli¬ 
cable  category  on  their  Contractor  Performance 

Evaluation  Survey 

Customer 

Satisfaction 

Accenture 

5:1  ROI  for  quality  activities 

Return  on 

Investment ' 

Northrop  Grum¬ 
man 

13:1  ROI  calculated  as  defects  avoided  per  hour 
spent  in  training  and  defect  prevention 

Return  on 

Investment 

Thales  TT&S 

! 

Processes  for  earlier  defect  detection,  improved 
risk  management,  and  better  project  control 
implemented  after  showing  positive  return  on 
investment  during  pilot 

Return  on 

Investment 

Cases  and 
Quotations 

“The  following  list  summarizes  the  results  from  the  12  cases  for  each  of  the  five  classes  of 
performance  measures: 

•  Cost:  Six  cases  provide  nine  examples  of  cost-related  benefits,  including  reductions 
in  the  cost  to  find  and  fix  a  defect,  and  overall  cost  savings. 
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Sample  Sites 
Described 


Sample  of 
Programs 
Supported 


Period  of  Study 


Publication  Date 


Source  Article 


•  Schedule:  Eight  cases  provide  evidence  of  schedule-related  benefits,  including  de¬ 
creased  time  needed  to  complete  tasks  and  increased  predictability  in  meeting  sched¬ 
ules. 

•  Quality:  Five  cases  provide  evidence  of  measurable  improvements  in  quality,  mostly 
related  to  reduction  of  defects  over  time  or  by  product  life  cycle. 

•  Customer  Satisfaction:  Three  cases  show  improvements  in  customer  satisfaction, 
including  demonstration  of  customer  satisfaction  through  award  fees. 

•  Return  on  Investment:  Three  cases  claim  positive  return  on  investment  from  their 

CMMI-based  process  improvement,  and  two  of  these  provided  the  actual  results  of 
their  calculations.’'  _ 


•  Accenture 

•  Boeing  Australia  Limited 

•  General  Motors  Corporation 

•  Lockheed  Martin  Management  and  Data  Systems 

•  Northrop  Grumman  Information  Technology 

•  Thales  Air  Traffic  Management 

•  Thales  Simulation  and  Training 

•  Thales  Research  and  Technology,  UK 

•  Bosch  Gasoline  Systems 

•  JP  Morgan  Chase  and  Company,  Investment  Bank  Technology 

•  Sanchez  Computer  Associates,  Inc.  _ 


(Omnibus  assessment  of  effects  without  reference  to  specific  programs) 


“Demonstrating  the  Impact  and  Benefits  of  CMMI®:  An  Update  and  Preliminary  Results,” 
Goldenson,  D.R.,  Gibson,  D.L.,  Software  Engineering  Institute,  CMU/SEI-2003-SR-009, 
October,  2003 


5.7  Space  and  Naval  Warfare  Systems  Center 

The  Space  and  Naval  Warfare  Systems  Center  (SSC-SD)  improved  their  SmartNet  scheduling 
tool  for  high  performance  computing  environments.  A  team  was  assembled;  its  members  were 
SSC-SD  and  SAIC  personnel,  consultants,  and  academic  collaborators  from  the  Naval  Post 
Graduate  School,  George  Mason  University,  University  of  Cincinnati,  and  Purdue.  Their  mission 
was  to  achieve  a  CMM  Maturity  Level  3  for  the  project,  produce  a  high-quality  product  with  high 
reliability,  while  maintaining  a  high  level  of  control  in  configuration  management.  Achieving 
their  goal  led  to  a  45%  decrease  in  software  change  requests  in  18  months. 
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The  Lesson:  The  standardized  language  and  Key  Process  Areas  of  the  CMM  enabled  disparate 
groups  to  work  together  toward  a  common  improvement  goal.  Almost  none  of  the  benefits 
achieved,  such  as  “better  communication,”  “improved  quality”  are  in  any  way  specific  to  a  devel¬ 
opment  environment. 


Table  9:  Space  and  Naval  Warfare  Systems  Center 


Organizational 
Study  Sponsor 

Space  and  Naval  Warfare  Systems  Center  (SSC-SD) 

Scope  of  Effort 

Improve  SmartNet  scheduling  tool  for  high-performance  computing  Environments. 

Improvement 

Focus  and 
Objectives 

Achieve  a  CMM  Maturity  Level  3  for  the  team/project,  and  produce  a  high-quality  product 
with  high  reliability,  while  maintaining  a  high  level  of  control  in  configuration  manage¬ 
ment. 

Method 

CMM,  along  with  informal  peer  and  design  reviews 

Sample  of 

Reported  Benefits 

Software  Process  Improvement  activities  associated  with  the  achievement  of  CMM  Level  3 
resulted  in  45%  reduction  in  the  number  of  software  change  requests  over  18  months,  as 

well  as: 

•  Better  overall  performance  of  the  software 

•  Better  documentation 

•  Software  delivery  closer  to  scheduled  date 

•  Higher  quality  software 

•  Higher  customer  satisfaction 

•  Improved  employee  morale 

•  Better  communication  among  the  team 

•  Less  overtime  required  to  get  the  job  done 

•  Employees  better  educated 

•  Increase  employee  pride  in  their  work  and  increased  responsibility  and  accountability 

for  their  work 

Cases  and 
Quotations 

SmartNet  Project  Manager  reported,  “Better  management  control.” 

“According  to  one  employee,  the  same  people  who  were  resistant  to  implementing  SPI 
activities  would  not  want  to  work  on  a  project  that  didn’t  practice  SPI  now.” 

Sample  Sites 
Described 

Software  Process  Improvement  activities  were  implemented  by  Heterogeneous  Computing 
Team  responsible  for  SmartNet.  Team  consisted  of  SSC-SC  personnel,  SAIC  personnel, 
Consultants,  Academic  collaborators  from  the  Naval  Post  Graduate  School,  George  Mason 
University,  University  of  Cincinnati,  and  Purdue.  In  total,  the  team  consisted  of  12  full-time 
and  seven  part-time  staff  members  plus  various  academic  collaborators. 

Sample  of 
Programs 
Supported 

SmartNet  is  being  used  by  the  NASA  Earth  Observation  System  (EOS),  the  National  Insti¬ 
tute  of  Health,  the  Joint  Task  Force  Advanced  Technology  Demonstration  project,  the  Na¬ 
tional  Security  Agency,  Navy  Simulation  Systems,  the  USS  Coronado,  and  Florida  State 
University,  among  others. 
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Period  of  Study 

1995-  1997 

Publication  Date 

December  1997 

Source  Article 

Costs  and  Benefits  of  Software  Process  Improvement,  Prenger,  Karen  D.,  Naval  Postgradu¬ 
ate  School,  Monterey,  CA,  December  1997 

5.8  Software  Productivity  Research 

Software  Productivity  Research  studied  four  test  program  set  development  projects  located  in  the 
Test  Software  Branches  of  the  Oklahoma  City  Air  Logistics  Center  (OC-ALC),  Directorate  of 
Aircraft  (LA),  Software  Division  (LAS).  The  study  measured  the  economic  benefit  of  software 
process  improvement  for  these  projects.  Using  CMM,  as  well  as  internally-developed  methods, 
LAS  improvement  efforts  led  to  a  7  to  1  ROI  and  a  savings  of  $11M  over  eight  years,  a  90%  re¬ 
duction  in  defect  rates  compared  to  the  baseline  project,  and  a  26%  reduction  in  average  cost  of 
maintenance  actions  over  24  months. 

The  Lesson:  While  CMM  was  used  as  the  benchmark,  it  did  not  drive  the  improvement  effort. 
CMM  was  supplemented  by  internally  developed  methods.  Additionally,  the  needs  of  the  organi¬ 
zation  drove  the  entire  improvement  effort,  rather  than  allowing  CMM  requirements  to  dictate 
improvement  activities.  And  once  again,  “defect  rates,”  “maintenance,”  and  “productivity”  are 
enterprise  generic  values  of  improvement. 


Table  10:  Oklahoma  City  Air  Logistics  Center 


Organizational 
Study  Sponsor 

Oklahoma  City  Air  Logistics  Center  (OC-ALC) 

Scope  of  Effort 

Directorate  of  Aircraft  (LA),  Software  Division  (LAS) 

Improvement 

Focus  and 
Objectives 

Goal  was  to  determine  the  economic  benefits  of  software  process  improvement. 

Method 

CMM 

Sample  of 

Reported 

Benefits 

Category 

Result 

Return  On  In¬ 
vestment 

7.5:1,  An  investment  of  $1.5M  saved  $1 1.3M  over  8  years 

Defect  Rates 

90%  reduction  from  baseline  project  to  second  project 

Maintenance 

Costs 

26%  reduction  in  average  cost  of  a  TPS  maintenance  action  over  the 

last  two  years 

Productivity 

10X  increase  in  source  code  produced  from  baseline  to  most  recent 
project  due  to  technology  as  well  as  CMM  improvements 
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Cases  and 
Quotations 

“One  key  to  LAS'  success  is  that  they  have  worked  hard  to  implement  a  process  improve¬ 
ment  infrastructure  that  responds  to  the  true  needs  of  the  organization.  The  SEI  CMM  is  used 
only  as  a  guide.  An  improvement  is  only  implemented  if  the  organization  sees  that  it  will  add 
value  to  the  processes.  Additionally,  many  improvements  that  could  not  be  directly  traced  to 
the  CMM  were  important  to  the  organization  and  had  major  impacts  on  the  improvement 

efforts.” 

Sample  Sites 
Described 

OC-ALC  is  located  at  Tinker  AFB,  OK.  The  Software  Division  consists  of  over  400  person¬ 
nel  in  7  branches.  The  study  focused  four  test  program  set  development  projects  located  in 
the  Test  Software  Branches. 

Sample  of 
Programs 
Supported 

•  C-141  All  Weather  Landing  System,  1986-1989 

•  B-1B  Electrical  Multiplexing  System,  1986-1988 

•  C-5B  Automatic  Flight  Control  System,  1987-1990 

•  F-l  10  Digital  Engine  Controller,  1992-1994 

Period  of  Study 

Four  projects  studied  took  place  between  1986  and  1995: 

Publication  Date 

1995 

Source  Article 

“The  Economic  Benefits  of  Software  Process  Improvement,”  Butler,  Kelly  L.,  CrossTalk, 

July,  1995 

5.9  Air  Force  Institute  of  Technology 

The  Air  Force  Institute  of  Technology  tracked  cost  and  schedule  data  for  two  centers:  the  Aero¬ 
nautical  Systems  Center  (ASC)  at  Wright-Patterson  AFB  and  the  Electronic  Systems  Center  at 
Hanscom  AFB.  Cost  and  schedule  data  covered  a  12-month  period.  Researchers  wanted  to  de¬ 
termine  the  link  between  CMM  levels  and  software  development  success.  Their  findings  docu¬ 
mented  the  fact  that  mature  organizations  (those  with  higher  CMM  levels)  were  likely  to  have  on- 
baseline  cost  and  schedule  performance.  Conversely,  those  organizations  that  were  least  mature 
were  likely  to  have  difficulty  adhering  to  cost  and  schedule  baselines. 


The  Lesson:  Figures  can  often  be  interpreted  different  ways  and,  especially  in  competitive  situa¬ 
tions,  can  be  suspect.  However  this  study  determined  that,  on  a  very  fundamental  level,  a  correla¬ 
tion  does  exist  between  process  maturity  and  software  development  success. 
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Table  11:  Air  Force  Institute  of  Technology 

Organizational 

Study  Sponsor 

Air  Force  Institute  of  Technology 

Scope  of  Effort 

Measured  software  process  improvement  data  that  met  the  following  criteria: 

•  Projects  tracked  software- specific  cost  and  schedule  data  in  the  Cost/Schedule  Con¬ 
trols  Systems  Criteria  (C/CSC)  format 

•  Contractors  were  rated  according  to  SEI CMM  protocols 

•  Relevance  of  cost  and  schedule  data  to  the  rating  could  established 

Improvement  Focus 
and  Objectives 

Determine  the  correlation  between  software  process  maturity  and  software  process  suc¬ 
cess,  as  measured  by  cost  and  schedule  indicators 

Method 

Correlational  study  of  CMM  as  an  improvement  vehicle  associated  with  cost  and  sched¬ 
ule  variance 

Sample  of  Reported 
Benefits 

•  Cost  Performance  Index.  The  multiple  comparison  test  showed  significant  differ¬ 
ences  between  CMM  Levels  1  and  2  and  between  CMM  Levels  I  and  3.  As  the  or¬ 
ganization  increases  in  CMM  maturity,  its  CPI  general  approaches  LOO  with  gener¬ 
ally  decreasing  variation. 

•  Schedule  Performance  Index.  At  all  CMM  rating  levels,  the  SPI  remains  close  to 

1.00.  However  for  projects  less  than  80%  complete,  the  performance  of  Level  1  or¬ 
ganizations  was  consistently  below  an  SPI  of  1 .00.  As  they  exceeded  the  80%  com¬ 
pleted  milestone,  it  increased  its  SPI  to  meet  the  required  1 .00 

Cases  and 

Quotations 

“The  aim  of  our  research  was  determine  the  nature  of  a  correlation  between  the  CMM 
rating  and  software  development  success...  We  observed  improved  cost  and  schedule 
performance  with  increasing  process  maturity.  Specifically,  the  least  mature  organiza¬ 
tions  were  likely  to  have  difficulty  adhering  to  cost  and  schedule  baselines.  In  contrast, 
the  more  mature  organizations  were  likely  to  have  on-baseline  cost  and  schedule  per¬ 
formance.” 

Sample  Sites 
Described 

•  Aeronautical  Systems  Center  (ASC)  Wright-Patterson  AFB 

•  Electronic  Systems  Center  (ESC)  Hanscom  AFB 

Sample  of  Programs 
Supported 

Data  on  1 1  DoD  contractors  who  had  been  rated  by  CMM  protocols  on  3 1  software  pro¬ 
jects  was  collected. 

Period  of  Study 

1993-  1994 

Publication  Date 

1995 

Source  Article 

“A  Correlational  Study  of  the  CMM  and  Software  Development  Performance,”  Lawlis, 

Dr.  Patricia  K.;  Flowe,  Robert  M.;  Thordahl  Capt.  James  B.,  Crosstalk ,  September,  1995 
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5.10  Motorola  Government  Electronics  Division 

The  Motorola  Government  Electronics  Division  (GED)  has  embedded  a  number  of  software  im¬ 
provement  methodologies  and  programs  as  part  of  its  corporate  culture.  These  include  CMM,  Six 
Sigma,  the  10X  cycle  time  initiative,  and  several  internally-developed  programs.  As  part  of  Mo¬ 
torola  GED’s  continuing  improvement  program,  each  of  34  projects  evaluates  key  process  areas 
every  quarter.  Metrics  for  each  project  were  compared  to  relevant  baseline  projects  between  1994 
and  1997.  Motorola  personnel  calculated  an  ROI  of  6.77  to  1  for  a  typical  100,000  source  lines 
of  code  (SLOC)  project.  Such  a  project  would  span  18  months  and  require  20  software  engineers. 

The  Lesson:  Process  improvement  activities  are  not  mutually  exclusive.  Depending  on  how  they 
are  used,  they  can  complement  each  other,  leading  to  greater  gains  and  returns  on  investment.  In 
this  case,  several  improvement  initiatives  were  employed  in  concert. 


Table  12:  Motorola  Government  Electronics  Division 


Organizational 

Study  Sponsor 

Motorola  Government  Electronics  Division 

Scope  of  Effort 

Motorola  GED  personnel  reviewed  34  projects  in  various  stages  of  completion  for  cycle 
time,  quality  and  productivity  metrics. 

Improvement  Focus 
and  Objectives 

As  part  of  Motorola  GED’s  continuing  improvement  program,  each  of  34  projects  evalu¬ 
ates  key  process  areas  every  quarter.  Metrics  for  each  project  were  compared  to  relevant 
baseline  projects. 

Method 

Primarily  CMM  supported  by: 

•  Six  Sigma  quality  focus 

•  10X  cycle-time  initiative 

•  Several  internally-developed  programs 

Sample  of  Reported 
Benefits 

•  ROI  of  6.77  to  1  for  a  typical  100,000  source  lines  of  code  (SLOC).  project.  Such  a 
project  would  span  18  months  and  require  20  software  engineers.  In  dollar  amounts, 
Motorola  saved  $611,200  after  investing  $90,180  in  a  typical  project. 

Cases  and 

Quotations 

“Each  level  of  CMM  maturity  achieved  cuts  defect  density  by  a  factor  of  two.  Cycle  time 
and,  to  a  lesser  extent,  productivity  improve  with  each  maturity  level  except  Level  3, 
when  cycle  time  and  productivity  decrease.  From  this  we  can  conclude  that  achieving 

Level  3  involves  significant  new  process  introduction,  which  can  negatively  affect  these 
two  metrics  until  the  project  absorbs  and  learns  to  tailor  the  processes.” 

Sample  Sites 
Described 

Motorola  GED  employs  350  engineers  in  software  development. 

Sample  of  Programs 
Supported 

Motorola  GED  designs  and  builds  a  wide  variety  of  government  electronic  systems 

Period  of  Study 

1994 _ 1997 

Publication  Date 

1997 
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Source  Article 

“How  Software  Process  Improvement  Helped  Motorola,”  Diaz,  Michael  and  Sligo,  Jo¬ 

seph,  IEEE  Software ,  Sept./Oct.,  1977 

5.1 1  Northrop  Grumman  Electronics  Systems  (ES) 

Northrop  Grumman  Electronics  Systems  (ES)  builds  air  traffic  control  systems,  sensors  for  the 
Defense  Meteorological  Support  Program,  radar  systems  for  the  FI  6  and  F22  fighters,  and  other 
advanced  components  of  defense  weapons  systems.  During  the  1996  to  2001  timeframe,  the 
company  was  engaged  in  obtaining  CMM  Level  3  recertification  and  moving  up  to  Level  4  using 
CMMI  and  internally-developed  methods.  Moving  from  Level  3  to  Level  4  resulted  in  20%  an¬ 
nual  gains  in  productivity  year  for  the  5-year  reporting  period.  This  represented  a  10%  gain  over 
its  previous  annual  improvement  achievement.  The  acceleration  resulted  in  a  $25  million  annual 
cost  avoidance  over  the  5-year  time  span. 

The  Lesson:  Cost  avoidance  doesn’t  necessarily  show  up  on  the  bottom  line.  However,  it  still 
represents  a  major  benefit  and  a  driving  force  behind  any  software  process  improvement  activity. 


Table  13:  Northrop  Grumman  Electronics  Systems 


Organizational 

Study  Sponsor 

Northrop  Grumman  Electronics  Systems  (ES) 

Scope  of  Effort 

Accelerate  productivity  gains  } 

Move  to  the  use  of  product  lines,  architecture  and  systematic  reuse 

Improvement  Focus 
and  Objectives 

Obtain  SEI  Capability  Maturity  Model  Level  3  recertification  and  move  up  to  Level  4 

Method 

The  process  group  consists  of  30-year  veterans,  senior  management,  technical  personnel, 
and  retired  line  managers.  They  formulated  a  plan  to  tie  process  improvement  to  business 
goals.  They  typically  spend  $2M  on  software  process  improvement.  They  used  CMMI 
plus  internally-developed  processes. 

Sample  of  Reported 
Benefits 

Moving  from  Level  3  to  Level  4  resulted  in  20%  annual  gains  in  productivity  year  for  the 
5-year  reporting  period.  This  represented  a  10%  gain  over  its  previous  annual  improve¬ 
ment  achievement.  The  acceleration  resulted  in  a  $25M  annual  cost  avoidance  over  the  5- 

year  time  span. 

Cases  and 

Quotations 

“Based  on  the  tangible  benefits  accrued,  Northrop  Grumman  ES  has  become  a  true  be¬ 
liever  in  process  improvement.  Either  accelerated  productivity  or  a  move  to  product  lines 
alone  would  have  justified  our  investments.  However,  the  real  scorecard  happens 
monthly  at  our  internal  financial  reviews.  Before  our  process  initiative,  we  used  to  spend 
hours  explaining  why  many  of  our  software  projects  had  problems  delivering  acceptable 
products  on  schedule  and  within  budget.  Today,  life  is  easier.  We  run  our  software  or¬ 
ganization  like  a  business.  Few  of  our  projects  are  in  trouble.  Yes,  there  are  still  chal¬ 
lenges  that  we  must  address.  But  we  aren’t  scolded  any  longer  for  being  the  problem  on 
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6  Summary 


During  the  course  of  reviewing  the  articles,  documents,  books,  and  Web  sites,  several  themes  be¬ 
came  apparent. 

1.  All  improvement  efforts  are  based  on  the  concept  and  the  goal  of  eliminating  errors  in  proc¬ 
ess  upstream. 

2.  While  improvement  efforts  are  typically  driven  by  cost  and  measured  by  return  on  invest¬ 
ment,  quality  attributes  such  as  security  and  stability  are  often  of  more  value  to  the  success 
of  the  overall  project. 

3.  Most  improvement  efforts  will  yield  benefits  as  long  as  those  efforts  follow  general  rules  and 
use  appropriate  methods. 

These  three  themes  form  a  framework  that  can  be  used  to  evaluate  process  improvement  activities 
at  large. 

This  review  of  many  varied  types  of  organizations  reveals  a  few  fundamental  observations  about 
improvement  efforts  in  general.  One  key  observation  is  that  a  process  is  a  process  is  a  process. 
This  is  not  to  say  that  all  organizations  are  the  same,  however,  all  organizations  employ  activities 
to  achieve  their  goals  and  objectives.  It  is  the  intelligent  manipulation  of  these  activities  in  spe¬ 
cific  ways  that  helps  organizations  to  change  for  the  better. 
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Feedback 


Please  send  comments  to  Peter  Capell  (psc@sei.cmu.edu). 
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